Deep learning, stochastic gradient descent and diffusion maps

Stochastic gradient descent (SGD) is widely used in deep learning due to its computational efficiency, but a complete understanding of why SGD performs so well remains major challenge. It has been observed empirically that most eigenvalues the Hessian loss functions on landscape over-parametrized neural networks are close zero, while only small number large. Zero indicate zero diffusion along corresponding directions. This indicates process minima selection mainly happens relatively low-dimensional subspace top Hessian. Although parameter space very high-dimensional, these findings seems dynamics may live manifold. In this paper, we pursue truly data driven approach problem getting potentially deeper high-dimensional surface, and particular, traced out by analyzing generated through SGD, or any other optimizer for matter, order possibly discover (local) representations optimization landscape. As our vehicle exploration, use maps introduced R. Coifman coauthors.